Image inventory production

ABSTRACT

There are disclosed methods and apparatus for manufacture of image inventories from frames of a digital work including audio which corresponds to objects in still images of the digital video work. Objects are detected in each frame&#39;s image, and the objects are recognized. Metadata is assigned to the objects, the object metadata linking audio from the digital video work to the corresponding object in the frame&#39;s image which produces the audio. For each frame, at least one cryptographic hash of the object metadata is generated, and the hash is written to a node of a transaction processing network.

RELATED APPLICATION INFORMATION

This patent application is a continuation-in-part of and claims priorityto the following prior-filed copending non-provisional patentapplications: U.S. application Ser. No. 16/017,876, entitled“MANUFACTURE OF PRINTED IMAGE INVENTORIES”, filed on Jun. 25, 2018; U.S.application Ser. No. 16/000,410, entitled “MANUFACTURE OF IMAGEINVENTORIES”, filed on Jun. 5, 2018; U.S. application Ser. No.16/027,068, entitled “PRODUCTION OF MODIFIED IMAGE INVENTORIES”, filedon Jul. 3, 2018; and U.S. application Ser. No. 16/110,831, entitled“MANUFACTURE OF INVENTORIES OF IMAGE PRODUCTS”, filed on Aug. 23, 2018;each of which are hereby incorporated by reference in their entirety.

NOTICE OF COPYRIGHTS AND TRADE DRESS

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. This patent document may showand/or describe matter which is or may become trade dress of the owner.The copyright and trade dress owner has no objection to the facsimilereproduction by anyone of the patent disclosure as it appears in thePatent and Trademark Office patent files or records, but otherwisereserves all copyright and trade dress rights whatsoever.

BACKGROUND Field

This disclosure relates to digital image processing, and moreparticularly to production of physical prints from digital images.

Description of the Related Art

A movie is a collection of still images that are shown one after theother in quick succession. When the images are viewed in quicksuccession (approximately 24 images per second), the human braininterprets the images as a motion picture or movie. Therefore, atraditional movie that is shown in theaters is a display of images inquick succession on the order of approximately 24 images per second, or129,600 images for a 90-minute movie (24 images per second×60 secondsper minute×90 minutes). Movies made in non-digital (e.g., photographicfilm) media can be converted to digital format, and the converted moviesas well as movies originally made in digital format can have individualimages extracted from the master digital media file. Physical (e.g.,hard copy) prints can be made from the still images. Artificialintelligence technology is applied to the content of the master digitalmedia file to improve the exposure of individual images, whereby theprinciples of sensitometry are used in training the artificialintelligence algorithms that are applied to improve the image exposure.Upon completion of the process of improvement of image exposure, afilter can be applied to convert the individual images to a desiredformat (such as where a sepia filter is applied to convert black andwhite or color images to a sepia and white format). Artificialintelligence technology is applied to the content of the master digitalmedia file to convert monochromatic (such as black and white) sourceimages into color-converted images The improved and converted images canbe assembled into a new, derivative movie work through linking theindividual images into a movie, using the image numbering metadata toplace each image in its respective order (e.g. image 2 follows image 1and precedes image 3 and so on) in a new, derivative digital media file.High-resolution (e.g., Ultra-High Definition (3840×2160 pixels) orhigher) video display technology enables the display of images withdetails such as texture (e.g., linen versus smooth and fine-grainedwoods) that are not possible with lower resolution (e.g., 1920×1080pixels).

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system for production of imageinventories.

FIG. 2 is a block diagram of a computing device.

FIG. 3 is a flowchart of a process for production of image inventories.

FIG. 4 is a block diagram of a frame slicing and packaging machine.

FIG. 5 is a block diagram of a system for production of printed imageinventories.

FIG. 6 is a block diagram of a computing device.

FIG. 7 is a flowchart of a process for production of physical printsfrom image inventories.

FIG. 8 is a block diagram of a physical print production and packagingmachine.

FIG. 9 is a block diagram of a system for production of imageinventories.

FIG. 10 is a block diagram of a computing device.

FIG. 11 is a flowchart of a process for production of image inventories.

FIG. 12 is a block diagram of a frame slicing and packaging machine.

FIG. 13 is a block diagram of a system for production of modified imageinventories having an improved image exposure and a monochromatic filterapplied to the original images resulting from the system of FIG. 9.

FIG. 14 is a flowchart of a process for production of modified imageinventories being processed by artificial intelligence technology toimprove the image exposure of the original source images followed byapplication of a filter to the images with improved exposure.

FIG. 15 is a block diagram of a system for production of modified imageinventories being processed by artificial intelligence technology toimprove the image exposure of the original source image inventories andthen apply a monochrome filter to the improved exposure imageinventories.

FIG. 16 is a flowchart of a process for assembling modified individualimage inventories into derivative movie works.

FIG. 17 is a block diagram of a system for assembling modifiedindividual image inventories into derivative movie works.

FIG. 18 is a block diagram of a system for production of modified imageinventories consisting of black and white images converted to colorimages.

FIG. 19 is a flowchart of a process for conversion of black and whiteimages to color image inventories through the application of artificialintelligence technology to the process of color conversion of the blackand white images.

FIG. 20 is a block diagram of a system for production of color-convertedimage inventories where artificial intelligence technology is used toperform the color-conversion process of converting black and whiteimages to color images.

FIG. 21 is a flowchart of a process for assembling color-converted imageinventories into derivative movie works.

FIG. 22 is a block diagram of a system for assembling color-convertedimage inventories into derivative movie works.

FIG. 23 is a block diagram of a system for production of inventories ofimage products.

FIG. 24 is a block diagram of a computing device.

FIG. 25 is a flowchart of a process for production of inventories ofimage products.

FIG. 26 is a block diagram of a frame slicing and packaging machine.

FIG. 27 is a block diagram of a system for production of derivativeimage products.

FIG. 28. is a flowchart of a process for production of derivative imageproducts.

FIG. 29 is a block diagram of a production and packaging machine.

Throughout this description, elements appearing in figures are assignedthree-digit or four-digit reference designators, where the mostsignificant digit(s) are the figure number and the two least significantdigits are specific to the element. An element that is not described inconjunction with a figure may be presumed to have the samecharacteristics and function as a previously-described element having areference designator with the same least significant digits.

DETAILED DESCRIPTION

Description of Apparatus

Referring now to FIG. 1 there is shown a block diagram of a system 100for production of image inventories. The system 100 includes one or morefilm libraries 110 a, 110 b, 110 c, a frame slicing and packagingmachine 120, an inventory of finished image products 130, andtransaction processing nodes 140.

The film libraries 110 a, 110 b, 110 c each include a collection of oneor more whole movies, videos, and/or movie trailers, and/or portionsthereof (e.g., clips), each of which is a sequence of frames having arespective still image. These items in a film library will be referredto herein as base works. A base work may be a single frame, i.e., astill image, such as a picture or a drawing, in analog or digitalformat. The base works may be in analog or digital format, and each filmlibrary 110 a, 110 b, 110 c may be exclusive to a particular form orformat of base work. Some film libraries may have base works in assortedforms and/or formats, related or unrelated. The frames of a base workmay have various objects, such as people, animals, goods physicalstructures or text in a frame. A given frame may include audio andsubtitles.

The frame slicing and packaging machine 120 produces the inventory offinished image products 130 and moves them into the transactionprocessing nodes 140. The frame slicing and packaging machine 120 may bea computer system, such as shown in FIG. 2, including one or morenon-volatile machine-readable media storing a program havinginstructions which when executed by a processor will cause the processorto produce the inventory of finished image products 130. As shown inFIG. 4, the frame slicing and packaging machine 120 may include a framemetadata unit 410, an object detection unit 420, an object recognitionunit 430, an object metadata unit 440, a hash unit 450, a watermark unit460, and a posting unit 470. These units 410, 420, 430, 440, 450, 460,470 interact with a work in process subsystem 490, which may be storage240 (FIG. 2). The posting unit 470 may effectuate its work through I/Ointerface 280 (FIG. 2).

Artificial intelligence may be incorporated into or used by the framemetadata unit 410, the object detection unit 420, the object recognitionunit 430 and/or the object metadata unit 440. Accordingly, these units410, 420, 430, 440 may be trained to perform the corresponding workprior to going into production. These units 410, 420, 430, 440 mayemploy quality assurance, such as use of human checks on samples of theproduction output, which may be used as feedback for refinement of thetraining.

Each finished image product in the inventory 130 is a chattel good,capable of transfer on an individual basis.

The transaction processing nodes 140 may be in an information technologycloud 150, such as cloud storage. The transaction processing nodes maybe in a blockchain.

Description of Processes

Referring now to FIG. 3 there is shown a flowchart 300 of a process forproduction of image inventories. The process 300 may be performed by theframe slicing and packaging machine 120 of FIG. 1 and FIG. 4, having asinputs the film libraries 110 a, 110 b, 110 c, etc. and manufacturingthe inventory of finished image products 130. The flow chart 300 hasboth a start 305 and an end 395, but the process is cyclical in nature.

After starting 305, the process may include some preprocessing. Thispreprocessing may include extraction of base works from the filmlibraries into the work in process subsystem 490, and extraction offrames from a base work. The base work may be on an analog physicalmedium such as celluloid film, and preprocessing may include scanningthe analog film medium into a digital file. If the base work is inanalog form it may be converted to digital form. Preprocessing resultsin the base work, in digital form, being stored in the work in processsubsystem 490.

During preprocessing, a human operator may select which frames should beincluded or excluded from further processing by the frame slicing andpackaging machine 120. Frame selection criteria may include metadata asdescribed below. The operator may be provided with options for frameselection, such as actors. For example, if the film Butch Cassidy andthe Sundance Kid was being preprocessed, the operator could choose toprocess only the frames having video and/or audio of Butch Cassidy(played by Paul Newman), or only the frames showing his face.

Next, metadata is assigned to each frame of the digital video work (step310). This may be performed by the frame metadata unit 410. The framemetadata may include an identification of the work, provenance of thework, an identification of the processor, and an identification of theframe within the work. The metadata may include: colors in the frame;tags previously assigned to the frame that describe the frame; andgeographic location represented in the frame. The provenance of the workmay include: the identity of the frame slicing and packaging machine;the geographic location and timestamp where the frame was originallyproduced or subsequently processed; names or other identification of thepeople, equipment and firms which did the production and processing;language and subtitles; details of how the production and processingwere performed; and details of errors and errata from the production andprocessing. Metadata for a given frame may include a sequence of imagesfrom immediate prior and or subsequent frames, or thumbnails of them,such as from a video or collection of still images. The metadata foreach frame may be stored in a single file, with a CSV, XML or JSONformat. The metadata from plural frames may be stored in a database, andthe database may also include the correlated images.

Metadata for frames of video may include: frame number in the overallframe count for the movie, segment, video, video clip, or trailer beingprocessed; frame number in a given segment or fragment of the whole basework; frame number in its relative position in a given segment orfragment of the base work; license attributes such as whether electronicreproduction and/or hardcopy printing are permitted; genre; category;title of the base work; title of the scene; starting time code; endingtime code; duration; frame count; producer; director; and studio.

Metadata for frames of still images may include: title or identity ofthe collection or sub-collection of which the image is a part; artist;subject; category; album; session; sequence; session index; sequenceindex; camera used; and number of photographs in the album, session andor sequence.

Next, objects in each frame are detected (step 320). This may beperformed by the object detection unit 420. The object detection stepmay include differentiation from a series of frames, for example using aKalman filter on the images to recognize objects in motion.

Next, the detected objects are recognized (step 330). This may beperformed by the object recognition unit 430. These objects may berecognizable generically or with varying specificity. For example, anobject may be recognized generically as a person, or as a specific typeof person (e.g., adult, child, male, female, star actor, extra), or as aspecific person (e.g., Mae West or John F. Kennedy). Recognition ofaudio objects may include speech to text conversion. Frames may includetext objects, such as signs or labels in an image.

Next, metadata is assigned to the recognized objects (step 340). Thismay be performed by the object metadata unit 440. This metadata mayinclude the location in the frame of the object and recognition of theobject (i.e., identification of what the object is). Location of theobject in the frame may be, for example, X-Y coordinates, whether theobject is in the frame's video or audio, bounding box coordinates (e.g.x0, y0, width, height), or foreground/background. The metadata mayinclude an image of the object (e.g., an actor). The metadata of aperson may include the actor's name. The metadata for audio objects mayinclude spoken lines and sounds.

The metadata may link objects from within frames or across frames. Forexample, audio may be linked to the object in the image which producesthe audio. In this way lines of dialogue may be linked to the actorspeaking the lines. For example, in Gone with the Wind, Rhett Butler(played by Clark Gable) may be linked to the line, “Frankly, my dear, Idon't give a damn.” Likewise, recitations of “May the Force be with you”in a Star Wars film may be linked to each actor reciting this line. Thesound of a gunshot may be linked to the image of the gun, or to a personstruck, such as in the movie The Matrix, when Neo (played by KeanuReeves) is shot.

Additional frame or object metadata may include: whether it is a heroshot, where a famous actor appears in the frame; lead actors, where leadactors who may not be hero level actors, but are still the lead actorsfor the movie, appear in the frame; other actors that are not leadactors appear in the frame; famous locations, such as Monument Valley,Ariz., appearing in the frame; popular or famous objects, such as theMillennium Falcon, appearing in the frame; desired color compositionappearing in the frame; quality of preservation or original sourcemedia, as whether it is deteriorated or is damaged; pre-existing valueof adjacent segments or frames.

Next, for each frame, a cryptographic hash is generated of the frame'simage, the frame metadata, and the object metadata (step 350). This maybe performed by the hash unit 450, which encrypts information itemsabout a frame along with the frame itself into a hash value thatuniquely identifies the frame and information items about the frame.Thus, if even a single digital bit is changed in the files of the frameand information items about the frame that are input into thecryptographic hashing algorithm, the resulting hash value will becompletely different from the hash value before the digital bit waschanged. The hash unit 450 may produce a hash value from therepresentation of the image file along with the frame and objectmetadata. This hash value is more efficient and technologically superiorto prior art index prints which aim to confirm that an individual printis part of a collection of prints shown in the index print.

The cryptographic hashing of the image from a frame with the metadata isperformed to be able to objectively and transparently confirmauthenticity of the image and the metadata into the future. This allowsreliable chain of title and chain of custody, and can support a reliablemarket for the image.

Next a watermark may be added to the hash (step 360) to protect thelinkage of the image file with its hashed value from malicioustampering. This may be performed by the watermark unit 460. Thewatermark unit 460 packages the hash value with the corresponding frame.

Next, the watermarked hash is written to a node of a transactionprocessing network (step 370). This may be performed by the posting unit470 by generating a transaction to register the watermarked hash alongwith its record provenance into a blockchain. Writing to the node may beunder control of a smart contract. The hash values provided by the hashunit 450, or the packages from the watermark unit 460, are recorded bythe posting unit 470 into the transaction processing nodes 140, whichmay be in a secure transaction processing network, distributed ledger orblockchain, or other transaction processing environment. The distributedledger may be an immutable distributed ledger.

The transaction processing nodes may support queries. Thus, anindividual who might want to purchase an image could make a query toconfirm the authenticity of the image. Each frame will have a differenthash value—even hashes of the same frame will have different hashvalues.

FIG. 2 is a block diagram of a computing device 200. A computing deviceas used herein refers to any device with a processor, memory and astorage device that may execute instructions including, but not limitedto, personal computers and server computers. These computing devices mayrun an operating system, including variations of the Linux, MicrosoftWindows, and Apple Mac operating systems.

The computing device 200 may be representative of the frame slicing andpackaging machine 120 (FIG. 1). The computing device 200 may includesoftware and/or hardware for providing functionality and featuresdescribed herein. The computing device 200 may therefore include one ormore of: logic arrays, memories, analog circuits, digital circuits,software, firmware and processors. The hardware and firmware componentsof the computing device 200 may include various specialized units,circuits, software and interfaces for providing the functionality andfeatures described herein. For example, a global positioning system(GPS) receiver or similar hardware may provide location-based services.

The computing device 200 has a processor 210 coupled to a memory 220,storage 240, a network interface 260 and an I/O interface 280. Theprocessor 210 may be or include one or more microprocessors, fieldprogrammable gate arrays (FPGAs), application specific integratedcircuits (ASICs), programmable logic devices (PLDs) and programmablelogic arrays (PLAs).

The memory 220 is a non-transitory storage medium and may be or includeRAM, ROM, DRAM, SRAM and MRAM, and may include firmware, such as staticdata or fixed instructions, BIOS, system functions, configuration data,and other routines used during the operation of the computing device 200and processor 210. The memory 220 also provides a storage area for dataand instructions associated with applications and data handled by theprocessor 210. As used herein the term memory corresponds to the memory220 and explicitly excludes transitory media such as signals orwaveforms. The techniques disclosed herein may be implemented withmachine readable storage media in a storage device included with orotherwise coupled or attached to a computing device. That is, thesoftware may be stored in electronic, machine readable media.

The storage 240 provides non-volatile, bulk or long-term storage of dataor instructions in the computing device 200. The storage 240 may takethe form of a magnetic or solid-state disk, tape, CD, DVD, or otherreasonably high capacity addressable or serial storage medium. Multiplestorage devices may be provided or available to the computing device200. Some of these storage devices may be external to the computingdevice 200, such as network storage or cloud-based storage. In somecases, such as those involving solid state memory devices, the memory220 and storage 240 may be a single device.

The network interface 260 includes an interface to a network such as anetwork that can be used to communicate calls, signals, streams, arrays,flagged samples and feedback described herein. The network interface 260may be wired or wireless.

The I/O interface 280 interfaces the processor 210 to peripherals (notshown) such as displays, video and still cameras, microphones, keyboardsand USB devices.

In some cases, storage 240 is a non-volatile machine-readable storagemedium that includes all types of computer readable media, includingmagnetic storage media, optical storage media, and solid-state storagemedia. The software can be installed in the frame slicing and packagingmachine 120.

The technologies described herein provide various technologicalimprovements to computer performance and efficiency. For example, theframe slicing and packaging machine 120 has performance enhancementsover the prior art that results in more efficient production of aninventory of image products from frames of a digital video work. Forexample, the technologies described are technological improvements overthose of the past because they provide verifiable provenance of imagesof frames that have been extracted from a motion picture, short videosuch as a music video, video clip, movie trailer, or individual stillphotographs. It is believed that there is no mechanism in the prior artto extract individual frames and concurrent with the extract, provideproof of authenticity or provenance of the extracted frames and metadataabout the extracted frames.

Referring now to FIG. 5 there is shown a block diagram of a system 500for production of physical print inventories. The system 500 includesone or more finished image product inventory 510 items, a physical printproduction and packaging machine 520, an inventory of finished physicalprint products 530, and transaction processing nodes 140.

The physical print production and packaging machine 520 receives imageproducts from the inventory of finished image products 130, produces theinventory of finished physical print products 530, and moves them intothe transaction processing nodes 140. The physical print production andpackaging machine 520 may be a computer system, such as shown in FIG. 6,including one or more non-volatile machine-readable media storing aprogram having instructions which when executed by a processor willcause the processor to produce the inventory of finished image products530.

As shown in FIG. 8, the physical print production and packaging machine520 may include a physical print metadata unit 810, a physical printframing (e.g., the decorative and/or protective device that the physicalprint is attached to or mounted in) metadata unit 820, a hash unit 850,a watermark unit 860, and a posting unit 870. These units 810, 820, 850,860, 870 interact with a work in process subsystem 890, which may bestorage 640 (FIG. 6). The posting unit 870 may effectuate its workthrough the I/O interface 680 (FIG. 6).

Each finished print product in the inventory 530 is a chattel good,capable of transfer on an individual basis.

The transaction processing nodes 140 may be in an information technologycloud 150, such as cloud storage. The transaction processing nodes maybe in a blockchain.

Description of Processes

Referring now to FIG. 7 there is shown a flowchart 700 of a process forproduction of physical prints. The process 700 may be performed by thephysical print production and packaging machine 520 of FIG. 5 and FIG.8, having as inputs the finished image product inventory 510 items, andmanufacturing the inventory of finished physical print products 530. Theflow chart 700 has both a start 705 and an end 795, but the process iscyclical in nature.

Next, metadata is assigned to each physical print of the digital videowork being produced (step 710). This may be performed by the printmetadata unit 810. The print metadata may include a copy of thefollowing items: identification of the work, provenance of the work, anidentification of the frame within the work, and may include additionalnew items. The metadata may include: colors in the print; effects orfilters added to the print; tags previously assigned to the image framethat describe the frame being printed; package elements of the printsuch as where an image is split into two or more sub-prints to give asense of motion; physical substrate material the print is printed on;where a physical frame or other protective holding device is orderedwith the print, the material the device is composed of; border material,if any; print protective materials; print and frame protectivematerials, if any; and hanging system supplied with the print, if any.The provenance of the work may include: the identity of the printproduction and packaging machine; the geographic location and timestampwhere the print was originally produced or subsequently processed; namesor other identification of the people, equipment and firms which did theproduction and processing; language and subtitles; details of how theproduction and processing were performed; and details of errors anderrata from the production and processing. Metadata for a given imageframe that is being printed may include a sequence of images fromimmediate prior and or subsequent frames, or thumbnails of them, such asfrom a video or collection of still images. The metadata for each framemay be stored in a single file, with a CSV, XML or JSON format. Themetadata from plural frames may be stored in a database, and thedatabase may also include the correlated images.

Metadata for prints of frames of video may include: frame number in theoverall frame count for the movie or segment or video or video clip ortrailer being processed; frame number in a given segment or fragment ofthe whole base work; frame number in its relative position in a givensegment or fragment of the base work; license attributes such as whetherelectronic reproduction and/or hardcopy printing are permitted; genre;category; title of the base work; title of the scene; starting timecode; ending time code; duration; frame count; producer; director; andstudio.

Metadata for prints of frames of still images may include: title oridentity of the collection or sub-collection of which the image is apart; artist; subject; category; album; session; sequence; sessionindex; sequence index; camera used; and number of photographs in thealbum, session and or sequence.

Additional frame or object metadata may include: whether it is a heroshot, where a famous actor appears in the frame; lead actors, where leadactors who may not be hero level actors, but are still the lead actorsfor the movie, appear in the frame; other actors that are not leadactors appear in the frame; famous locations, such as Monument Valley,Ariz., appearing in the frame; popular or famous objects, such as theMillennium Falcon, appearing in the frame; desired color compositionappearing in the frame; quality of preservation or original sourcemedia, as whether it is deteriorated or is damaged; and pre-existingvalue of adjacent segments or frames.

Next, for each frame, a cryptographic hash is generated of the frame'simage, the frame metadata, and the object metadata (step 750). This maybe performed by the hash unit 850, which encrypts information itemsabout a print along with the digital file of the print itself into ahash value that uniquely identifies the printed image frame andinformation items about the image frame. Thus, if even a single digitalbit is changed in the files of the print and information items about theprint that are input into the cryptographic hashing algorithm, theresulting hash value will be completely different from the hash valuebefore the digital bit was changed. The hash unit 850 may produce a hashvalue from the representation of the print file along with the imageframe and object metadata. The hash unit 850 packages the hash valuewith the corresponding print. This hash value is more efficient andtechnologically superior to prior art index prints which aim to confirmthat an individual print is part of a collection of prints shown in theindex print.

The cryptographic hashing of the image from a frame with the metadata isperformed to be able to objectively and transparently confirmauthenticity of the image and the metadata into the future. This allowsreliable chain of title and chain of custody and can support a reliablemarket for the image.

Next a watermark may be added to the hash (step 760) to protect thelinkage of the image file with its hashed value from malicioustampering. This may be performed by the watermark unit 860. Thewatermark unit 860 packages the watermarked hash value with thecorresponding print.

Next, the watermarked hash is written to a node of a transactionprocessing network (step 770). This may be performed by the posting unit870 by generating a transaction to register the watermarked hash alongwith its record provenance into a blockchain. Writing to the node may beunder control of a smart contract. The hash values provided by the hashunit 850, or the packages from the watermark unit 860, are recorded bythe posting unit 870 into the transaction processing nodes 140, whichmay be in a secure transaction processing network, distributed ledger orblockchain, or other transaction processing environment. The distributedledger may be an immutable distributed ledger.

The transaction processing nodes may support queries. Thus, anindividual who might want to purchase an image inventory item 130 or aphysical print item 530, with the physical print item 530 as a packagewith a protective or decorative frame, or just the physical print, couldmake a query to confirm the authenticity of the image. Each hashedpackage or physical print will have a different hash value—even hashesof the same package or physical print will have different hash values.

FIG. 6 is a block diagram of a computing device 600. A computing deviceas used herein refers to any device with a processor, memory and astorage device that may execute instructions including, but not limitedto, personal computers and server computers. These computing devices mayrun an operating system, including variations of the Linux, MicrosoftWindows, and Apple Mac operating systems.

The computing device 600 may be representative of the physical printproduction and packaging machine 520 (FIG. 5). The computing device 600may include software and/or hardware for providing functionality andfeatures described herein. The computing device 600 may thereforeinclude one or more of: logic arrays, memories, analog circuits, digitalcircuits, software, firmware and processors. The hardware and firmwarecomponents of the computing device 600 may include various specializedunits, circuits, software and interfaces for providing the functionalityand features described herein. For example, a global positioning system(GPS) receiver or similar hardware may provide location-based services.

The computing device 600 has a processor 610 coupled to a memory 620,storage 640, a network interface 660 and an I/O interface 680. Theprocessor 610 may be or include one or more microprocessors, fieldprogrammable gate arrays (FPGAs), application specific integratedcircuits (ASICs), programmable logic devices (PLDs) and programmablelogic arrays (PLAs).

The memory 620 is a non-transitory storage medium and may be or includeRAM, ROM, DRAM, SRAM and MRAM, and may include firmware, such as staticdata or fixed instructions, BIOS, system functions, configuration data,and other routines used during the operation of the computing device 600and processor 610. The memory 620 also provides a storage area for dataand instructions associated with applications and data handled by theprocessor 610. As used herein the term memory corresponds to the memory620 and explicitly excludes transitory media such as signals orwaveforms. The techniques disclosed herein may be implemented withmachine readable storage media in a storage device included with orotherwise coupled or attached to a computing device. That is, thesoftware may be stored in electronic, machine readable media.

The storage 640 provides non-volatile, bulk or long-term storage of dataor instructions in the computing device 600. The storage 640 may takethe form of a magnetic or solid-state disk, tape, CD, DVD, or otherreasonably high capacity addressable or serial storage medium. Multiplestorage devices may be provided or available to the computing device600. Some of these storage devices may be external to the computingdevice 600, such as network storage or cloud-based storage. In somecases, such as those involving solid-state memory devices, the memory620 and storage 640 may be a single device.

The network interface 660 includes an interface to a network such as anetwork that can be used to communicate calls, signals, streams, arrays,flagged samples and feedback described herein. The network interface 660may be wired or wireless.

The I/O interface 680 interfaces the processor 610 to peripherals (notshown) such as displays, video and still cameras, microphones, keyboardsand USB devices.

In some cases, storage 640 is a non-volatile machine-readable storagemedium that includes all types of computer readable media, includingmagnetic storage media, optical storage media, and solid-state storagemedia. The software can be installed in the frame slicing and packagingmachine 520.

The technologies described herein provide various technologicalimprovements to computer performance and efficiency. For example, theframe slicing and packaging machine 520 has performance enhancementsover the prior art that results in more efficient production of aninventory of image products from frames of a digital video work. Forexample, the technologies described are technological improvements overthose of the past because they provide verifiable provenance of imagesof frames that have been extracted from a motion picture, short videosuch as a music video, video clip, movie trailer, or individual stillphotographs. It is believed that there is no mechanism in the prior artto extract individual frames and concurrent with the extract, provideproof of authenticity or provenance of the extracted frames and metadataabout the extracted frames.

Referring now to FIG. 9 there is shown a block diagram of a system 900for production of image inventories. The system 900 includes one or morefilm libraries 110 a, 110 b, 110 c, a frame slicing and packagingmachine 920, an inventory of finished image products 930, andtransaction processing nodes 140.

The film libraries 110 a, 110 b, 110 c each includes a collection of oneor more whole movies, videos, and/or movie trailers, and/or portionsthereof (e.g., clips), each of which is a sequence of frames having arespective still image. These items in a film library will be referredto herein as base works. A base work may be a single frame, i.e., astill image, such as a picture or a drawing, in analog or digitalformat. The base works may be in analog or digital format, and each filmlibrary 110 a, 110 b, 110 c may be exclusive to a particular form orformat of base work. Some film libraries may have base works in assortedforms and/or formats, related or unrelated. The frames of a base workmay have various objects, such as people, animals, goods or text in aframe. A given frame may include audio and subtitles.

The frame slicing and packaging machine 920 produces the inventory offinished image products 930 and moves them into the transactionprocessing nodes 140. The frame slicing and packaging machine 920 may bea computer system, such as shown in FIG. 10, including one or morenon-volatile machine-readable media storing a program havinginstructions which when executed by a processor will cause the processorto produce the inventory of finished image products 930. As shown inFIG. 12, the frame slicing and packaging machine 920 may include a framemetadata unit 1210, an object detection unit 1220, an object recognitionunit 1230, an object metadata unit 1240, a hash unit 1250, a watermarkunit 1260, a posting unit 1270. These units 1210, 1220, 1230, 1240,1250, 1260, 1270 interact with a work in process subsystem 1290, whichmay be the storage 1040 (FIG. 10). The posting unit 1270 may effectuateits work through the I/O interface 1080 (FIG. 10).

Artificial intelligence may be incorporated into or used by the framemetadata unit 1210, the object detection unit 1220, the objectrecognition unit 1230 and/or the object metadata unit 1240. Accordingly,these units 1210, 1220, 1230, 1240 may be trained to perform thecorresponding work prior to going into production. These units 1210,1220, 1230, 1240 may employ quality assurance, such as use of humanchecks on samples of the production output, which may be used asfeedback for refinement of the training.

Referring to FIG. 13 there is shown a block diagram of a system 1300 forproduction of modified image inventories having the image exposureimproved by application of artificial intelligence technology to theinventory of finished image products 930 of FIG. 9. The system 1300includes one or more finished image products 930, a frame conversion andpackaging machine 1320, an inventory of modified derivative videolibraries 1330, an inventory of modified derivative image products 1310,and transaction processing nodes 140.

The inventory of finished image products 930 each are processed byartificial intelligence technology to improve the image exposure,whereby the principles of sensitometry are used to train the artificialintelligence algorithms that are applied to improve the image exposure.Upon completion of improvement of image exposure, a monochromatic filtermay be applied to the exposure-improved images to further alter theoriginal image format to a new format such as sepia and white, or toimprove the image contrast such as by adding a yellow filter to theimage, or other artistic alterations to the image such as antiquing,burning, dodging and the like.

The frame conversion and packaging machine 1320 produces the inventoryof modified image products 1310 and inventory of modified video products1330 and moves them into the transaction processing nodes 140. The frameconversion and packaging machine 1320 may be a computer system, such asshown in FIG. 10, including one or more non-volatile machine-readablemedia storing a program having instructions which when executed by aprocessor will cause the processor to produce the inventory of modifiedimage products 1310 or modified video products 1330. As shown in FIG.15, the frame conversion and packaging machine 1320 may include a framemeta image processing unit 1510, an image exposure smoothing unit 1530,an improved image exposure metadata unit 1540, a monochrome imageenhancement unit 1550, a hash unit 1560, a watermark unit 1570, and aposting unit 1580. These units 1510, 1530, 1540, 1550, 1560, 1570, 1580interact with a work in process subsystem 1590, which may be the storage1040 (FIG. 10). The posting unit 1580 may effectuate its work throughthe I/O interface 1080 (FIG. 10). Alternatively, as shown in FIG. 17,the frame conversion and packaging machine 1320 may include a frameimage sequencing unit 1710, a metadata unit 1720, a hash unit 1730, awatermark unit 1740, and a posting unit 1750. These units 1710, 1720,1730, 1740, 1750 interact with a work in process subsystem 1760, whichmay be the storage 1040 (FIG. 10). The posting unit 1750 may effectuateits work through the I/O interface 1080 (FIG. 10).

Artificial intelligence may be incorporated into or used by the framemeta image processing unit 1510, the frame image exposure smoothing unit1530, and/or the improved image exposure metadata unit 1540.Accordingly, these units 1510, 1530, 1540 may be trained to perform thecorresponding work prior to going into production. These units 1510,1530, 1540 may employ quality assurance, such as use of human checks onsamples of the production output, which may be used as feedback forrefinement of the training.

The inventory of modified image products 1310 can be assembled into avideo through use of the metadata attribute of image number. A video canbe assembled by organizing a group of converted images according totheir relative order through use of the metadata attribute of imagenumber as an organizing index (e.g. image 1 precedes image 2 which isfollowed by image 3) and the organized converted images are then writtento a new derivative video digital media file in the inventory ofmodified video products 1330.

Referring to FIG. 18 there is shown a block diagram of a system 1800 forproduction of color-converted image inventories where monochromatic(e.g. black and white) source images are converted to color images byapplication of artificial intelligence technology to the inventory offinished image products 930 of FIG. 9. The system 1800 includes one ormore finished image products 930, a frame color-conversion and packagingmachine 1820, an inventory of color-converted derivative video libraries1830, an inventory of color-converted derivative image products 1810,and transaction processing nodes 140.

The inventory of finished image products 930 each are processed byartificial intelligence technology to convert the source image fromblack and white to color. In this context, objects in the image aredetected, recognized and colored by artificial intelligence technologyusing algorithms that have been developed through extensive trainingiterations. The use of artificial intelligence technology to perform thecolorization effort can result in significant increases in accuracy ofcolorized result while realizing a corresponding significant reductionin the time it takes to perform the color-conversion of the sourceimage.

The frame color-conversion and packaging machine 1820 produces theinventory of color-converted image products 1810 and inventory ofcolor-converted video products 1830 and moves them into the transactionprocessing nodes 140. The frame color-conversion and packaging machine1820 may be a computer system, such as shown in FIG. 10, including oneor more non-volatile machine-readable media storing a program havinginstructions which when executed by a processor will cause the processorto produce the inventory of color-converted image products 1810 orcolor-converted video products 1830. As shown in FIG. 20, the framecolor-conversion and packaging machine 1820 may include a frame objectdetection unit 2010, an object recognition and colorization unit 2020,an object metadata unit 2030, a hash unit 2040, a watermark unit 2050,and a posting unit 2060. These units 2010, 2020, 2030, 2040, 2050, 2060interact with a work in process subsystem 2090, which may be the storage1040 (FIG. 10). The posting unit 2060 may effectuate its work throughthe I/O interface 1080 (FIG. 10). Alternatively, as shown in FIG. 22,the frame color-conversion and packaging machine 1820 may include acolor-converted image sequencing unit 2210, a metadata unit 2220, a hashunit 2230, a watermark unit 2240, and a posting unit 2270. These units2210, 2220, 2230, 2240, 2270 interact with a work in process subsystem2290, which may be the storage 1040 (FIG. 10). The posting unit 2270 mayeffectuate its work through the I/O interface 1080 (FIG. 10).

The inventory of color-converted image products 1810 each can beassembled into a video through use of the metadata attribute of imagenumber. A video can be assembled by organizing a group ofcolor-converted images according to their relative order through use ofthe metadata attribute of image number as an organizing index (e.g.image 1 precedes image 2 which is followed by image 3) and the organizedcolor-converted images are then written to a new derivative videodigital media file 1830.

Each finished image product in the inventory 930, 1310, 1330, 1810 and1830 is a chattel good, capable of transfer on an individual basis.

The transaction processing nodes 140 may be in an information technologycloud 150, such as cloud storage. The transaction processing nodes maybe in a blockchain.

Description of Processes

Referring now to FIG. 11 there is shown a flowchart 1100 of a processfor production of image inventories. The process 1100 may be performedby the frame slicing and packaging machine 920 of FIG. 9 and FIG. 12,having as inputs the film libraries 110 a, 110 b, 110 c, etc. andmanufacturing the inventory of finished image products 930. The flowchart 1100 has both a start 1105 and an end 1195, but the process iscyclical in nature.

After starting 1105, the process may include some preprocessing. Thispreprocessing may include extraction of base works from the filmlibraries into the work in process subsystem 1290, and extraction offrames from a base work. The base work may be on an analog physicalmedium such as celluloid film, and preprocessing may include scanningthe analog film medium into a digital file. If the base work is inanalog form it may be converted to digital form. Preprocessing resultsin the base work, in digital form, being stored in the work in processsubsystem 1290.

During preprocessing, a human operator may select which frames should beincluded or excluded from further processing by the frame slicing andpackaging machine. Frame selection criteria may include metadata asdescribed below. The operator may be provided with options for frameselection, such as actors. For example, if the film Butch Cassidy andthe Sundance Kid was being preprocessed, the operator could choose toprocess only the frames having Butch Cassidy (played by Paul Newman), oronly the frames showing his face.

Next, metadata is assigned to each frame of the digital video work (step1110). This may be performed by the frame metadata unit 1210. The framemetadata may include an identification of the work, provenance of thework, an identification of the processor, and an identification of theframe within the work. The metadata may include: colors in the frame;tags previously assigned to the frame that describe the frame; andgeographic location represented in the frame. The provenance of the workmay include: the identity of the frame slicing and packaging machine;the geographic location and timestamp where the frame was originallyproduced or subsequently processed; names or other identification of thepeople, equipment and firms which did the production and processing;language and subtitles; details of how the production and processingwere performed; and details of errors and errata from the production andprocessing. Metadata for a given frame may include a sequence of imagesfrom immediate prior and or subsequent frames, or thumbnails of them,such as from a video or collection of still images. The metadata foreach frame may be stored in a single file, with a CSV, XML or JSONformat. The metadata from plural frames may be stored in a database, andthe database may also include the correlated images.

Metadata for frames of video may include: frame number in the overallframe count for the movie or segment or video or video clip or trailerbeing processed; frame number in a given segment or fragment of thewhole base work; frame number in its relative position in a givensegment or fragment of the base work; license attributes such as whetherelectronic reproduction and/or hardcopy printing are permitted; genre;category; title of the base work; title of the scene; starting timecode; ending time code; duration; frame count; producer; director; andstudio.

Metadata for frames of still images may include: title or identify ofthe collection or sub-collection of which the image is a part; artist;subject; category; album; session; sequence; session index; sequenceindex; camera used; and number of photographs in the album, session andor sequence.

Next, objects in each frame are detected (step 1120). This may beperformed by the object detection unit 1220. The object detection stepmay include differentiation from a series of frames, for example using aKalman filter on the images to recognize objects in motion.

Next, the detected objects are recognized (step 1130). This may beperformed by the object recognition unit 1230. These objects may berecognizable generically or with varying specificity. For example, anobject may be recognized generically as a person, or as a specific typeof person (e.g., adult, child, male, female, star actor, extra), or as aspecific person (e.g., Mae West or John F. Kennedy). Recognition ofaudio objects may include speech to text conversion. Frames may includetext objects, such as signs or labels in an image.

Next, metadata is assigned to the recognized objects (step 1140). Thismay be performed by the object metadata unit 1240. This metadata mayinclude the location in the frame of the object and recognition of theobject (i.e., identification of what the object is). The metadata mayinclude an image of the object (e.g., an actor). The metadata of aperson may include the actor's name. The metadata for audio objects mayinclude spoken lines and sounds.

The metadata may link objects from within frames or across frames. Forexample, audio may be linked to the object in the image which producesthe audio. In this way lines of dialogue may be linked to the actorspeaking the lines. In this way, in Gone with the Wind, Rhett Butler(played by Clark Gable) may be linked to the line, “Frankly, my dear, Idon't give a damn.” Likewise, recitations of “May the Force be with you”in a Star Wars film may be linked to each actor reciting this line. Thesound of a gunshot may be linked to the image of the gun, or to a personstruck, such as in the move The Matrix, when Neo (played by KeanuReeves) is shot.

Additional frame or object metadata may include: whether it is a heroshot, where a famous actor appears in the frame; lead actors, where leadactors who may not be hero level actors, but are still the lead actorsfor the movie, appear in the frame; other actors that are not leadactors appear in the frame; famous locations, such as Monument Valley,Ariz., appearing in the frame; popular or famous objects, such as theMillennium Falcon, appearing in the frame; desired color compositionappearing in the frame; quality of preservation or original sourcemedia, as whether it deteriorated or is damaged; pre-existing value ofadjacent segments or frames.

Next, for each frame, a cryptographic hash is generated of the frame'simage, the frame metadata, and the object metadata (step 1150). This maybe performed by the hash unit 1250, which encrypts information itemsthat uniquely identify a frame into a hash value that uniquelyidentifies the frame's image and information items about the frame'simage. Thus, if even a single digital bit is changed in the files of theframe's image and information items about the frame's image that areinput into the cryptographic hashing algorithm, the resulting hash valuewill be completely different from the hash value before the digital bitwas changed. The hash unit 1250 may produce a hash value from theauthenticated representation of the image file along with the frame andobject metadata. This hash value is more efficient and technologicallysuperior to prior art index prints which aim to confirm that anindividual print is part of a collection of prints shown in the indexprint.

The cryptographic hashing of the image from a frame with the metadata isperformed to be able to objectively and transparently confirmauthenticity of the image and the metadata into the future. This allowsreliable chain of title and chain of custody and can support a reliablemarket for the image.

Next a watermark may be added to the hash (step 1160) to protect thelinkage of the image file with its hashed value from malicioustampering. This may be performed by the watermark unit 1260. Thewatermarking unit 1260 packages the hash value with the correspondingframe.

Next, the watermarked hash is written to a node of a transactionprocessing network (step 1170). This may be performed by the postingunit 1270 by generating a transaction to register the watermarked hashalong with its record provenance into a blockchain. Writing to the nodemay occur under control of a smart contract. The hash values provided bythe hash unit 1250, or the packages from the watermarking unit 1260, arerecorded by the posting unit 1270 into the transaction processing nodes140, which may be in a secure transaction processing network,distributed ledger or blockchain, or other transaction processingenvironment. The distributed ledger may be an immutable distributedledger.

Referring to FIG. 13 there is shown a flowchart 1300 of a process forproduction of modified image inventories being processed by artificialintelligence technology to improve the image exposure of the originalsource images followed by optional application of a monochromatic filterto the images that have had their exposure improved. The process 1300may be performed by the modified image inventory packaging machine 1320of FIG. 13 and FIG. 15, having as inputs the inventory of finished imageproducts 930 and manufacturing the modified image inventories 1310. Theflow chart 1400 has both a start 1405 and an end 1495, but the processis cyclical in nature.

After starting 1405, the process creates meta image files 1410, 1420 and1430 that are samples of the finished image products 930 with differentexposures applied to the image. The process of creating meta image filesmay include copying the finished image products 930 into the work inprocess subsystem 1590 of FIG. 15. The set of meta image files is storedin the work in process subsystem 1590.

The next step in the process involves initial artificial intelligenceprocessing of the meta images into a baseline meta image 1440 which isroughly analogous to an average of the exposure values applied to metaimages of steps 1410, 1420 and 1430 with no weighting for exposure. Thebaseline meta image of step 1440 is stored in the work in processsubsystem 1590 of FIG. 15.

After the baseline meta image of step 1440 has been created, theartificial intelligence exposure adjustment process moves to aniterative exposure adjustment between the three meta images where a copyof the baseline meta image is shifted between meta image exposuresettings until the range of exposures in the image is optimized based onalgorithms taught to the artificial intelligence processing subsystem1450, e.g., to smooth the image exposure. This step can be performed bythe image exposure smoothing unit 1530. A further process of applicationof a monochromatic (e.g. sepia) tint to the optimized-exposure modifiedimage can be processed at this stage by the monochrome image enhancementunit 1550. The resulting optimized-exposure modified image is stored inthe work in process subsystem 1590 of FIG. 15.

Next, metadata is assigned to each optimized-exposure modified image.This may be performed by the image metadata unit 1540. The imagemetadata may include an identification of the range of exposure valuesthat were used in the optimized-exposure modified image. Where amonochromatic tint was applied to the optimized-exposure modified image,the specific tint value would be included in the assigned metadata. Themetadata for each optimized-exposure modified image may be stored in asingle file, with a CSV, XML or JSON format. The metadata from pluraloptimized-exposure modified images may be stored in a database, and thedatabase may also include the correlated images.

Next, for each optimized-exposure modified image, a cryptographic hashis generated of the image and image metadata. This may be performed bythe hash unit 1560, which encrypts information items that uniquelyidentify the modified image and image metadata into a hash value thatuniquely identifies the modified image and information items about themodified image. Thus, if even a single digital bit is changed in thefiles of the modified image and information items about the modifiedimage that are input into the cryptographic hashing algorithm, theresulting hash value will be completely different from the hash valuebefore the digital bit was changed. The hash unit 1560 may produce ahash value from the authenticated representation of theoptimized-exposure modified image file along with the image metadata.This hash value is more efficient and technologically superior to priorart index prints which aim to confirm that an individual print is partof a collection of prints shown in an index print.

The cryptographic hashing of the modified image with the image metadatais performed to be able to objectively and transparently confirmauthenticity of the modified image and the metadata into the future.This allows reliable chain of title and chain of custody and can supporta reliable market for the image.

Next, a watermark may be added to the hash (step 1480) to protect thelinkage of the image file with its hashed value from malicioustampering. This may be performed by the watermark unit 1570. Thewatermark unit 1570 packages the hash value with the correspondingimage.

Next, the watermarked modified image hash is written to a node of atransaction processing network (step 1490). This may be performed by theposting unit 1580 by generating a transaction to register thewatermarked hash along with its record of provenance into a blockchain.Writing to the node may occur under control of a smart contract. Thehash values provided by the hash unit 1560, or the packages from thewatermark unit 1570, are recorded by the posting unit 1580 into thetransaction processing nodes 140, which may be in a secure transactionprocessing network, distributed ledger or blockchain, or othertransaction processing environment. The distributed ledger may be animmutable distributed ledger.

Referring to FIG. 16 there is shown a flowchart 1600 of a process forproduction of modified derivative video works having as inputs themodified image inventories 1310 and manufacturing the modifiedderivative video works 1330. The process 1600 may be performed by theframe conversion and packaging machine 1320 of FIG. 13 and FIG. 17,having as inputs the modified image inventories 1310 and manufacturingthe modified derivative video works 1330. The flow chart 1600 has both astart 1605 and an end 1655, but the process is cyclical in nature.

After starting 1605, the process creates an indexed modified imagecollection by organizing the modified image inventory by image numbermetadata where image number 1 precedes image number 2, which in turn isfollowed by image number 3 and so on until the entire modified imagecollection is sequenced according to image number (step 1610). Theprocess of creating the sequenced modified image collection may includecopying the modified image files into the work in process subsystem 1760of FIG. 17. The set of sequenced modified image files is stored in thework in process subsystem 1760.

Next, metadata is assigned to the sequenced modified images file (step1620). This may be performed by the metadata unit 1720. The sequencedimage collection metadata may include the endorsement of the modifiedwork by the owner of original work's copyright, the range (e.g. 1through 135,000) of images that have been sequenced into the video, thedate and time the sequencing occurred, the location where the sequencingoccurred and other items of information relevant to the manufacture ofthe modified video work. The metadata for each sequenced modified imagemay be stored in a single file, with a CSV, XML or JSON format. Themetadata from plural sequenced modified images may be stored in adatabase, and the database may also include the correlated images.

Next for each sequenced modified images file, a cryptographic hash isgenerated of the images and the image metadata (step 1630). This may beperformed by the hash unit 1730, which encrypts information items thatuniquely identify the sequenced modified images and image metadata intoa hash value that uniquely identifies the sequenced modified images andinformation items about the sequenced modified images. Thus, if even asingle digital bit is changed in the files of the sequenced modifiedimages and information items about the sequenced modified images thatare input into the cryptographic hashing algorithm, the resulting hashvalue will be completely different from the hash value before thedigital bit was changed. The hash unit 1730 may produce a hash valuefrom the authenticated representation of the sequenced modified imagesalong with the image metadata. This hash value is more efficient andtechnologically superior to prior art index prints which aim to confirmthat an individual print is part of a collection of prints shown in anindex print. The cryptographic hashing of the sequenced modified imageswith the image metadata is performed to be able to objectively andtransparently confirm authenticity of the sequence of modified imagesand the metadata into the future. This allows reliable chain of titleand chain of custody and can support a reliable market for the modifiedmovies.

Next, a watermark may be added to the hash (step 1640) to protect thelinkage of the sequenced modified image file with its hashed value frommalicious tampering. This may be performed by the watermark unit 1740.The watermark unit 1740 packages the hash with the correspondingsequenced modified image file.

Next, the watermarked sequenced modified image hash is written to a nodeof a transaction processing network (step 1650). This may be performedby the posting unit 1750 by generating a transaction to register thewatermarked hash along with its record of provenance into a blockchain.Writing to the node may occur under control of a smart contract. Thehash values provided by the hash unit 1730, or the packages from thewatermark unit 1740, are recorded by the posting unit 1750 into thetransaction processing nodes 140, which may be in a secure transactionprocessing network, distributed ledger or blockchain, or othertransaction processing environment. The distributed ledger may be animmutable distributed ledger.

Referring to FIG. 19, there is shown a flowchart 1900 of a process forproduction of color-converted image inventories being processed byartificial intelligence technology to detect objects in the images,recognize and colorize the objects in the images and assign metadata tothe recognized and colorized objects in the images. The process 1900 maybe performed by the frame color-conversion and packaging machine 1820 ofFIG. 18 and FIG. 20, having as inputs the inventory of finished imageproducts 930 and manufacturing the color-converted image productinventories 1810. The flow chart 1900 has both a start 1905 and an end1965, but the process is cyclical in nature.

After starting 1905, the process uses artificial intelligence to detectobjects in the image 1910. The process of detecting objects may includecopying the finished image products 930 into the work in processsubsystem 2090 of FIG. 20. The set of detected objects image files arestored in the work in process subsystem 2090.

The next step in the process uses artificial intelligence to recognizeand colorize 1920 the objects detected in step 1910, which may beperformed by the object recognition and colorization unit 2020. The setof colorized objects that are the result of 1920 performing itsprocessing are stored in the work in process subsystem 2090 of FIG. 20.

Next, metadata is assigned to each colorized image object that wasprocessed by the image color-conversion subsystem. This may be performedby the image metadata unit 2030. The image metadata may include anidentification of the images that were detected, the color(s) that wereassigned to the detected images and any other actions that wereperformed by the artificial intelligence technology in processing thecolor conversion functions. The metadata for each color converted imagemay be stored in a single file, with a CSV, XML or JSON format. Themetadata from plural color-converted images may be stored in a database,and the database may also include the correlated images.

Next, for each color-converted image, a cryptographic hash is generatedof the image and image metadata (step 1940). This may be performed bythe hash unit 2040, which encrypts information items that uniquelyidentifies the color-converted image and image metadata into a hashvalue. Thus, if even a single digital bit is changed in the files of thecolor-converted image and image metadata that are input into thecryptographic hashing algorithm, the resulting hash value will becompletely different from the hash value before the digital bit waschanged. The hash unit 2040 may produce a hash value from theauthenticated representation of the color-converted image file alongwith the image metadata. This hash value is more efficient andtechnologically superior to prior art index prints which aim to confirmthat an individual print is part of a collection of prints shown in anindex print.

The cryptographic hashing of the color-converted image with the imagemetadata is performed to be able to objectively and transparentlyconfirm authenticity of the modified image and the metadata into thefuture. This allows reliable chain of title and chain of custody and cansupport a reliable market for the image.

Next, a watermark may be added to the hash (step 1950) to protect thelinkage of the image file with its hashed value from malicioustampering. This may be performed by the watermark unit 2050. Thewatermark unit 2050 packages the hash value with the correspondingimage.

Next, the watermarked color-converted image is written to a node of atransaction processing network (step 1960). This may be performed by theposting unit 2060 by generating a transaction to register thewatermarked hash along with its record of provenance into a blockchain.Writing to the node may occur under control of a smart contract. Thehash values provided by the hash unit 2040, or the packages from thewatermark unit 2050, are recorded by the posting unit 2060 into thetransaction processing nodes 140, which may be in a secure transactionprocessing network, distributed ledger or blockchain or othertransaction processing environment. The distributed ledger may be animmutable distributed ledger.

Referring to FIG. 21 there is shown a flowchart 2100 of a process forproduction of color-converted video works having as inputs thecolor-converted image inventories 1810 and manufacturing thecolor-converted derivative video works 1830. The process 2100 may beperformed by the color-conversion and packaging machine 1820 of FIG. 18and FIG. 20, having as inputs the color-converted image inventories 1810and manufacturing the color-converted derivative video works 1830. Theflow chart 2100 has both a start 2105 and an end 2155, but the processis cyclical in nature.

After starting 2105, the process creates an indexed image collection byorganizing the color-converted image inventory by image number metadatawhere image number 1 precedes image number 2, which in turn is followedby image number 3 and so on until the entire color-converted imagecollection is sequenced according to image number (step 2110). Theprocess of creating the sequenced color-converted image collection mayinclude copying the color-converted image files into the work in processsubsystem 2290 of FIG. 22. The set of sequenced color-converted imagefiles is stored in the work in process subsystem 2290.

Next, metadata is assigned to the sequenced color-converted images file(step 2120). This may be performed by the metadata unit 2220. Thesequenced image collection metadata may include the endorsement of thecolor-converted work by the owner of the original work's copyright, therange (e.g. 1 through 135,000) of images that have been sequenced intothe video, the date and time the sequencing occurred, the location wherethe sequencing occurred and other items of information relevant to themanufacture of the color-converted video work. The metadata or eachsequenced color-converted image may be stored in a single file, with aCSV, XML or JSON format. The metadata from plural sequencedcolor-converted images may be stored in a database, and the database mayalso include the correlated images.

Next for each sequenced color-converted images file, a cryptographichash is generated of the images and the image metadata (step 2130). Thismay be performed by the hash unit 2230, which encrypts information itemsthat uniquely identify the sequenced color-converted images and imagemetadata into a hash value. Thus, if even a single digital bit ischanged in the files of the print and information items about thesequenced color-converted images and image metadata that are input intothe cryptographic hashing algorithm, the resulting hash value will becompletely different from the hash value before the digital bit waschanged. The hash unit 2230 may produce a hash value from theauthenticated representation of the sequenced color-converted imagesalong with the image metadata. This hash value is more efficient andtechnologically superior to prior art index prints which aim to confirmthat an individual print is part of a collection of prints shown in anindex print. The cryptographic hashing of the sequenced color-convertedimages with the image metadata is performed to be able to objectivelyand transparently confirm authenticity of the sequence ofcolor-converted images and the metadata into the future. This allowsreliable chain of title and chain of custody and can support a reliablemarket for the color-converted movies.

Next, a watermark may be added to the hash (step 2140) to protect thelinkage of the sequenced color-converted image file with its hashedvalue from malicious tampering. This may be performed by the watermarkunit 2240. The watermark unit 2240 packages the hash with thecorresponding sequenced color-converted image file.

Next, the watermarked sequenced color-converted image hash is written toa node of a transaction processing network (step 2150). This may beperformed by the posting unit 2270 by generating a transaction toregister the watermarked hash along with its record of provenance into ablockchain. Writing to the node may occur under control of a smartcontract. The hash values provided by the hash unit 2230, or thepackages from the watermark 2240, are recorded by the posting unit 2270into the transaction processing nodes 140, which may be in a securetransaction processing network, distributed ledger or blockchain orother transaction processing environment. The distributed ledger may bean immutable distributed ledger.

The transaction processing nodes may support queries. Thus, anindividual who might want to purchase an image could make a query toconfirm the authenticity of the image.

FIG. 10 is a block diagram of a computing device 1000. A computingdevice as used herein refers to any device with a processor, memory anda storage device that may execute instructions including, but notlimited to, personal computers and server computers. These computingdevices may run an operating system, including variations of the Linux,Microsoft Windows, and Apple Mac operating systems.

The computing device 1000 may be representative of the frame slicing andpackaging machine 920 (FIG. 9), the frame conversion and packagingmachine 1320, and/or the frame conversion and packaging machine 1820.The computing device 1000 may include software and/or hardware forproviding functionality and features described herein. The computingdevice 1000 may therefore include one or more of: logic arrays,memories, analog circuits, digital circuits, software, firmware andprocessors. The hardware and firmware components of the computing device1000 may include various specialized units, circuits, software andinterfaces for providing the functionality and features describedherein. For example, a global positioning system (GPS) receiver orsimilar hardware may provide location-based services.

The computing device 1000 has a processor 1010 coupled to a memory 1020,storage 1040, a network interface 1060 and an I/O interface 1080. Theprocessor 1010 may be or include one or more microprocessors, fieldprogrammable gate arrays (FPGAs), application specific integratedcircuits (ASICs), programmable logic devices (PLDs) and programmablelogic arrays (PLAs).

The memory 1020 is a non-transitory storage medium and may be or includeRAM, ROM, DRAM, SRAM and MRAM, and may include firmware, such as staticdata or fixed instructions, BIOS, system functions, configuration data,and other routines used during the operation of the computing device1000 and processor 1010. The memory 1020 also provides a storage areafor data and instructions associated with applications and data handledby the processor 1010. As used herein the term memory corresponds to thememory 1020 and explicitly excludes transitory media such as signals orwaveforms. The techniques disclosed herein may be implemented withmachine readable storage media in a storage device included with orotherwise coupled or attached to a computing device. That is, thesoftware may be stored in electronic, machine readable media.

The storage 1040 provides non-volatile, bulk or long-term storage ofdata or instructions in the computing device 1000. The storage 1040 maytake the form of a magnetic or solid-state disk, tape, CD, DVD, or otherreasonably high capacity addressable or serial storage medium. Multiplestorage devices may be provided or available to the computing device1000. Some of these storage devices may be external to the computingdevice 1000, such as network storage or cloud-based storage. In somecases, such as those involving solid-state memory devices, the memory1020 and storage 1040 may be a single device.

The network interface 1060 includes an interface to a network such as anetwork that can be used to communicate calls, signals, streams, arrays,flagged samples and feedback described herein. The network interface1060 may be wired or wireless.

The I/O interface 1080 interfaces the processor 1010 to peripherals (notshown) such as displays, video and still cameras, microphones, keyboardsand USB devices.

In some cases, storage 1040 is a non-volatile machine-readable storagemedium that includes all types of computer readable media, includingmagnetic storage media, optical storage media, and solid-state storagemedia. It should be understood that the software can be installed in theframe slicing and packaging machine 920.

The technologies described herein provide various technologicalimprovements to computer performance and efficiency. For example, theframe slicing and packaging machine 920 has performance enhancementsover the prior art that results in more efficient production of aninventory of image products from frames of a digital video work. Forexample, the technologies described are technological improvements overthose of the past because they provide verifiable provenance of imagesof frames that have been extracted from a motion picture, short videosuch as a music video, video clip, movie trailer, or individual stillphotographs. It is believed that there is no mechanism in the prior artto extract individual frames and concurrent with the extract, provideproof of authenticity or provenance of the extracted frames and metadataabout the extracted frames.

Referring now to FIG. 23 there is shown a block diagram of a system 2300for production of inventories of image products. The system 2300includes one or more film libraries 110 a, 110 b, 110 c, a frame slicingand packaging machine 2320, an inventory of image products 2330, and oneor more transaction processing nodes 140.

The film libraries 110 a, 110 b, 110 c each include a collection of oneor more whole movies, videos, and/or movie trailers, and/or portionsthereof (e.g., clips), each of which is a sequence of frames having arespective still image. These items in a film library will be referredto herein as a base work. A base work may be a single frame, i.e., astill image, such as a picture or a drawing, in analog or digitalformat. The base works may be in analog or digital format, and each filmlibrary 110 a, 110 b, 110 c may be exclusive to a particular form orformat of base work. Some film libraries may have base works in assortedforms and/or formats, related or unrelated.

The frames of a base work may have various objects, such as people,animals, goods, physical structures or text in a frame. A given framemay include audio and subtitles.

The frame slicing and packaging machine 2320 produces the inventory ofimage products 2330 and moves them into the transaction processing nodes140. The frame slicing and packaging machine 2320 may be a computersystem, such as shown in FIG. 24, including one or more non-volatilemachine-readable media storing a program having instructions which whenexecuted by a processor will cause the processor to produce theinventory of image products 2330. As shown in FIG. 26, the frame slicingand packaging machine 2320 may include a frame metadata unit 2610, anobject detection unit 2620, an object recognition unit 2630, an objectmetadata unit 2640, a hash unit 2650, a watermark unit 2660, a postingunit 2670. These units 2610, 2620, 2630, 2640, 2650, 2660, 2670 interactwith a work in process subsystem 2690, which may be the storage 2440(FIG. 24). The posting unit 2670 may effectuate its work through the I/Ointerface 2480 (FIG. 24).

Artificial intelligence may be incorporated into or used by the framemetadata unit 2610, the object detection unit 2620, the objectrecognition unit 2630 and/or the object metadata unit 2640. Accordingly,these units 2610, 2620, 2630, 2640 may be trained to perform thecorresponding work prior to going into production. These units 2610,2620, 2630, 2640 may employ quality assurance, such as use of humanchecks on samples of the production output, which may be used asfeedback for refinement of the training.

Each image product in the inventory 2330 is a chattel good, capable oftransfer on an individual basis.

The transaction processing nodes 140 may be in an information technologycloud 150, such as cloud storage. The transaction processing nodes 140may be in a blockchain.

Referring now to FIG. 25 there is shown a flowchart 2500 of a processfor production of image inventories. The process 2500 may be performedby the frame slicing and packaging machine 2320 of FIG. 23 and FIG. 26,having as inputs the film libraries 110 a, 110 b, 110 c, etc. andmanufacturing the inventory of image products 2330. The flow chart 2500has both a start 2505 and an end 2595, but the process is cyclical innature.

After starting 2505, the process may include some preprocessing. Thispreprocessing may include extraction of base works from the filmlibraries into the work in process subsystem 2690, and extraction offrames from a base work. The base work may be on an analog physicalmedium such as celluloid film, and preprocessing may include scanningthe analog film medium into a digital file. If the base work is inanalog form it may be converted to digital form. Preprocessing resultsin the base work, in digital form, being stored in the work in processsubsystem 2690.

During preprocessing, a human operator may select which frames should beincluded or excluded from further processing by the frame slicing andpackaging machine. Frame selection criteria may include metadata aboutthe frames. The operator may be provided with options for frameselection, such as actors. For example, if the film Butch Cassidy andthe Sundance Kid was being preprocessed, the operator could choose toprocess only the frames having video and/or audio of Butch Cassidy(played by Paul Newman), or only the frames showing his face.

Next, metadata is assigned to each frame of the digital video work (step2510). This may be performed by the frame metadata unit 2610. The framemetadata may include an identification of the work, provenance of thework, an identification of the processor, and an identification of theframe within the work. The metadata may include: colors in the frame;tags previously assigned to the frame that describe the frame; andgeographic location represented in the frame. The provenance of the workmay include: the identity of the frame slicing and packaging machine;the geographic location and timestamp where the frame was originallyproduced or subsequently processed; names or other identification of thepeople, equipment and firms which did the production and processing;language and subtitles; details of how the production and processingwere performed; and details of errors and errata from the production andprocessing. Metadata for a given frame may include a sequence of imagesfrom immediate prior and or subsequent frames, or thumbnails of them,such as from a video or collection of still images. The metadata foreach frame may be stored in a single file, with a CSV, XML or JSONformat. The metadata from plural frames may be stored in a database, andthe database may also include the correlated images.

Metadata for frames of video may include: frame number in the overallframe count for the movie or segment or video or video clip or trailerbeing processed; frame number in a given segment or fragment of thewhole base work; frame number in its relative position in a givensegment or fragment of the base work; license attributes such as whetherelectronic reproduction and/or hardcopy printing are permitted; genre;category; title of the base work; title of the scene; starting timecode; ending time code; duration; frame count; producer; director;and/or studio.

Metadata for frames of still images may include: title or identity ofthe collection or sub-collection of which the image is a part; artist;subject; category; album; session; sequence; session index; sequenceindex; camera used; and number of photographs in the album, sessionand/or sequence.

Next, objects in each frame are detected (step 2520). This may beperformed by the object detection unit 2620. The object detection stepmay include differentiation from a series of frames, for example using aKalman filter on the images to recognize objects in motion.

Next, the detected objects are recognized (step 2530). This may beperformed by the object recognition unit 2630. These objects may berecognizable generically or with varying specificity. For example, anobject may be recognized generically as a person, or as a specific typeof person (e.g., adult, child, male, female, star actor, extra), or as aspecific person (e.g., Mae West or John F. Kennedy). Recognition ofaudio objects may include speech to text conversion. Frames may includetext objects, such as signs or labels in an image.

Next, metadata is assigned to the recognized objects (step 2540). Thismay be performed by the object metadata unit 2640. This metadata mayinclude the location in the frame of the object and recognition of theobject (i.e., identification of what the object is). The metadata mayinclude an image of the object (e.g., an actor). The metadata of aperson may include the actor's name. The metadata for audio objects mayinclude spoken lines and sounds.

The metadata may link objects from within frames or across frames. Forexample, audio may be linked to the object in the image which producesthe audio. In this way lines of dialogue may be linked to the actorspeaking the lines. In this way, in Gone with the Wind, Rhett Butler(played by Clark Gable) may be linked to the line, “Frankly, my dear, Idon't give a damn.” Likewise, recitations of “May the Force be with you”in a Star Wars film may be linked to each actor reciting this line. Or,the sound of a gunshot may be linked to the image of the gun, or to aperson struck, such as in the movie The Matrix, when Neo (played byKeanu Reeves) is shot.

Additional frame or object metadata may include: whether it is a heroshot, where a famous actor appears in the frame; lead actors, where leadactors who may not be hero level actors, but are still the lead actorsfor the movie, appear in the frame; other actors that are not leadactors appear in the frame; famous locations, such as Monument Valley,Ariz., appearing in the frame; popular or famous objects, such as theMillennium Falcon, appearing in the frame; desired color compositionappearing in the frame; quality of preservation or original sourcemedia, as whether it is deteriorated or is damaged; and/or pre-existingvalue of adjacent segments or frames.

Next, for each frame, a cryptographic hash is generated of the frame'simage, the frame metadata, and the object metadata to produce the imageproduct (step 2550). This may be performed by the hash unit 2650, whichencrypts information items about a frame along with the frame itselfinto a hash value that uniquely identifies the frame and informationitems about the frame. Thus, if even a single digital bit is changed inthe files of the frame and information items about the frame that areinput into the cryptographic hashing algorithm, the resulting hash valuewill be completely different from the hash value before the digital bitwas changed. The hash unit 2650 may produce a hash value from therepresentation of the image file along with the frame and objectmetadata. This hash value is more efficient and technologically superiorto prior art index prints which aim to confirm that an individual printis part of a collection of prints shown in the index print.

The cryptographic hashing of the image from a frame with the metadata isperformed to be able to objectively and transparently confirmauthenticity of the image and the metadata into the future. This allowsreliable chain of title and chain of custody tracking and can support areliable market for the image.

Next a watermark may be added to the hash (step 2560) to protect thelinkage of the image file with its hashed value from malicioustampering. This may be performed by the watermark unit 2660. Thewatermark unit 2660 packages the hash value with the correspondingframe.

Next, the image product (i.e., the hash without the watermark) iswritten to a node of a transaction processing network (step 2570). Thismay be performed by the posting unit 2670 by generating a transaction toregister the watermarked hash along with its record provenance into ablockchain. Writing to the node may be under control of a smartcontract. The hash values provided by the hash unit 2650, or thepackages from the watermark unit 2660, are recorded by the posting unit2670 into the transaction processing nodes 140, which may be in a securetransaction processing network, distributed ledger or blockchain, orother transaction processing environment. The distributed ledger may bean immutable distributed ledger.

The image products, as stored into nodes in the transaction processingnetwork, have images which are exactly the same as the base works.However, after an image product has been produced and saved to thetransaction processing node 140, it may be desirable to modify theimage's appearance. That is, it may be desirable to produce a new imageproduct in which the image is a derivative of an image in an existingimage product. As used herein, an image is a derivative if itsappearance is dominated by the appearance of the (prior) image fromwhich it was derived. The derivative is either a transformation,modification or adaptation of the prior image. The new image product maylikewise be saved to a transaction processing node. Because the basis ofthe new image product is an existing image product, the image in theexisting image product may be a base work or itself a derivative.

Referring now to FIG. 27, there is shown a block diagram of a system2700 for production of derivative image inventories. The system 2700includes one or more image product inventories 2330, an inventory ofderivation options 2760, a production and packaging machine 2720, aninventory of derivative image products 2730, and transaction processingnodes 140.

The inventory of derivation options 2760 may include a number ofcategories: mats, borders (akin to a physical frame around a physicalpainting), edits and filters. Each category may include a number ofoptions. Options may be inclusive or exclusive, and exclusivity may beas to certain options, options having certain characteristics, or alloptions.

As with physical framed art, mat treatments can be applied between theborder and the image. All elements of physical mat products such ascolorization, textures, line treatments, coatings and the like can beapplied. Further, different mats can be specified for a given imagebased on external criteria such as seasons, duration of sunlight orother localized environmental considerations. Three-dimensional effectscan be replicated by the introduction of a shadowed treatment betweenthe image and the mat to give a sense of the image being mounted abovethe mat surface.

Borders can have all of the attributes of physical frames. Differentmaterials such as resin and varieties of woods, such as cherry, oak,walnut, bamboo and the like, can be digitally represented. Frametreatments such as antiquing, patina enhancement, high gloss,semi-gloss, matte and other finishes, bleaching and similar effects canbe represented in the image. As with mats, different border types andsizes can be specified for a given image based on external criteria,such as seasons, duration (or absence) of sunlight or other localizedenvironmental considerations.

Edits may include cropping, resizing, zooming, dodging, burning androtating. Cropping typically is performed to remove objectionablecontent from an image, while resizing is used to ‘fit’ an image that hasbeen cropped into a desired size. Zooming is generally used to emphasizea particular image element or elements. Dodging and burning aregenerally used to emphasize or de-emphasize image elements. Variationson the dodging and burning can be used with color images to keep one ormore desired features colorized, while converting the remainder of theimage into a black and white scheme. Rotation of an image can be used tomake images that were taken in one orientation (e.g., portrait orlandscape) and present them in another orientation to better fit in withsurrounding physical constraints, such as available wall space or to fiton an easel of a certain size.

Antiquing or other filters can be applied to the surface texturecaptured in the image of the matte border that is being added to theunderlying digital image. Overlays can be applied to give a sense ofthree-dimensional treatment or to produce a limited dodging or burningeffect which in turn can serve to emphasize or diminish an imageelement.

Note that multiple image enhancements (such as two or more borders thatcan consist of different color borders, different matte materials ortextures, different virtual frame materials and different monochromatictreatment that can be applied to the respective image enhancements) canbe added together into a combined enhanced image product. The multipleimage enhancements can be displayed individually under programmaticcontrol (e.g. cherry wood virtual frame plus beige linen surface matteborder for a period of 30 days and then a new combination such as walnutwood virtual frame plus pink smooth surface matte border for a period offifteen days with further combinations being able to be defined by theowner of the high-resolution video display technology device), or theycan be randomly presented on the high-resolution video display device.All of these enhancements can be identified through metadata attributesthat are attached to the metadata file that is created for theunderlying digital image.

The production and packaging machine 2720 produces the inventory ofderivative image products 2730 and moves them into the transactionprocessing nodes 140. The production and packaging machine 2720 may be acomputer system, such as shown in FIG. 24, including one or morenon-volatile machine-readable media storing a program havinginstructions which when executed by a processor will cause the processorto produce the inventory of derivative image products 2730.

As shown in FIG. 29, the production and packaging machine 2720 mayinclude a derivation unit 2910, a metadata unit 2920, a hash unit 2930,a watermark unit 2940, and a posting unit 2970. These units 2910, 2920,2930, 2940, 2970 interact with a work in process subsystem 2990, whichmay be the storage 2440 (FIG. 24). The posting unit 2970 may effectuateits work through the I/O interface 2480 (FIG. 24).

Each derivative image product in the inventory 2330 is a chattel good,capable of transfer on an individual basis.

The transaction processing nodes 140 may be the same or different fromthose in FIG. 23, and may be in the same or a different informationtechnology cloud 150. The transaction processing nodes may be in ablockchain.

Referring now to FIG. 28 there is shown a flowchart 2800 of a processfor production of derivative image products. The process 2800 may beperformed by the production and packaging machine 2720 of FIG. 27,having as inputs image products 2330 and manufacturing the inventory ofderivative image products 2730. The flow chart 2800 has both a start2805 and an end 2895, but the process is cyclical in nature.

After starting 2805, image products may be selected from the inventory2330 for derivation or for exclusion from further processing by theproduction and packaging machine 2720 (step 2810). Selection may be by ahuman operator and/or automated. The operator may be provided withoptions for selection of image products, such as actors.

A human operator may also select derivations to be applied to theselected image products (step 2820). For example, different border woodtreatments (e.g., cherry, walnut, oak and bamboo) could be selected fora given border. Based on season, one border wood treatment (cherry,walnut, oak, or bamboo) would be displayed with the image. As the seasonchanged, the border wood selection would likewise change.

The derivations are then applied to the image (step 2830). Derivationsare included in the combined image file and each particular derivationwould be displayed under local application software program control.This may be performed by the derivation unit 2910.

Next, metadata relating to the derivation is assigned to the derivativeimage (step 2840). This may be performed by the metadata unit 2920. Themetadata may include one or more metadata items from the un-derivativestill image. New items of metadata may define or identify thederivations and provenance. The provenance of the work may include: theidentity of the production and packaging machine; the geographiclocation and timestamp of the derivation(s); names or otheridentification of the people, equipment and firms which did thederivation production and processing; language and subtitles apart fromthose in the underlying image; details of how the production andprocessing were performed; and details of errors and errata from theproduction and processing. The metadata for each frame may be stored ina single file, with a CSV, XML or JSON format.

Next, a cryptographic hash is generated of the derivative image, theunderlying image metadata, and the derivative image metadata (step2850). This may be performed by the hash unit 2930, which encryptsinformation items about a derivative image along with the digital fileof the derivative image itself into a hash value that uniquelyidentifies the derivative image and information items about thederivative image. Thus, if even a single digital bit is changed in thefiles of the derivative image and information items about the derivativeimage that are input into the cryptographic hashing algorithm, theresulting hash value will be completely different from the hash valuebefore the digital bit was changed. The hash unit 2930 may produce ahash value from the representation of the derivative image file alongwith the metadata. The hash unit 2930 packages the hash value with thecorresponding derivative image.

The cryptographic hashing of the derivative image is performed to beable to objectively and transparently confirm authenticity of thederivative image and the metadata into the future. This allows reliablechain of title and chain of custody and can support a reliable marketfor the derivative image.

Next a watermark may be added to the hash (step 2860) to protect thelinkage of the derivative image file with its hashed value frommalicious tampering. This may be performed by the watermark unit 2940.The watermark unit 2940 packages the watermarked hash value with thecorresponding derivative digital image.

Next, the watermarked hash is written to a node of a transactionprocessing network (step 2870). This may be performed by the postingunit 2970 by generating a transaction to register the watermarked hashalong with its record provenance into a blockchain. Writing to the nodemay be under control of a smart contract. The hash values provided bythe hash unit 2930, or the packages from the watermark unit 2940, arerecorded by the posting unit 2970 into the transaction processing nodes140, which may be in a secure transaction processing network,distributed ledger or blockchain, or other transaction processingenvironment. The distributed ledger may be an immutable distributedledger.

The transaction processing nodes may support queries. Thus, anindividual who might want to purchase an image product or a derivativeimage product, could make a query to confirm its authenticity.

FIG. 24 is a block diagram of a computing device 2400. A computingdevice as used herein refers to any device with a processor, memory anda storage device that may execute instructions including, but notlimited to, personal computers and server computers. These computingdevices may run an operating system, including variations of the Linux,Microsoft Windows, and Apple Mac operating systems.

The computing device 2400 may be representative of the frame slicing andpackaging machine 2320 (FIG. 23) and the production and packagingmachine 2720 (FIG. 27). The computing device 2400 may include softwareand/or hardware for providing functionality and features describedherein. The computing device 2400 may therefore include one or more of:logic arrays, memories, analog circuits, digital circuits, software,firmware and processors. The hardware and firmware components of thecomputing device 2400 may include various specialized units, circuits,software and interfaces for providing the functionality and featuresdescribed herein. For example, a global positioning system (GPS)receiver or similar hardware may provide location-based services.

The computing device 2400 has a processor 2410 coupled to a memory 2420,storage 2440, a network interface 2460 and an I/O interface 2480. Theprocessor 2410 may be or include one or more microprocessors, fieldprogrammable gate arrays (FPGAs), application specific integratedcircuits (ASICs), programmable logic devices (PLDs) and programmablelogic arrays (PLAs).

The memory 2420 is a non-transitory storage medium and may be or includeRAM, ROM, DRAM, SRAM and MRAM, and may include firmware, such as staticdata or fixed instructions, BIOS, system functions, configuration data,and other routines used during the operation of the computing device2400 and processor 2410. The memory 2420 also provides a storage areafor data and instructions associated with applications and data handledby the processor 2410. As used herein the term memory corresponds to thememory 2420 and explicitly excludes transitory media such as signals orwaveforms. The techniques disclosed herein may be implemented withmachine readable storage media in a storage device included with orotherwise coupled or attached to a computing device. That is, thesoftware may be stored in electronic, machine readable media.

The storage 2440 provides non-volatile, bulk or long-term storage ofdata or instructions in the computing device 2320 and 2720. The storage2440 may take the form of a magnetic or solid-state disk, tape, CD, DVD,or other reasonably high capacity addressable or serial storage medium.Multiple storage devices may be provided or available to the computingdevice 2400. Some of these storage devices may be external to thecomputing device 2400, such as network storage or cloud-based storage.In some cases, such as those involving solid-state memory devices, thememory 2420 and storage 2440 may be a single device.

The network interface 2460 includes an interface to a network such as anetwork that can be used to communicate calls, signals, streams, arrays,flagged samples and feedback described herein. The network interface2460 may be wired or wireless.

The I/O interface 2480 interfaces the processor 2410 to peripherals (notshown) such as displays, video and still cameras, microphones, keyboardsand USB devices.

In some cases, storage 2440 is a non-volatile machine-readable storagemedium that includes all types of computer readable media, includingmagnetic storage media, optical storage media, and solid-state storagemedia. The software can be installed in the frame slicing and packagingmachine 2320.

The technologies described herein provide various technologicalimprovements to computer performance and efficiency. For example, theframe slicing and packaging machine 2320 has performance enhancementsover the prior art that results in more efficient production of aninventory of image products from frames of a digital video work. Forexample, the technologies described are technological improvements overthose of the past because they provide verifiable provenance of imagesof frames that have been extracted from a motion picture, short videosuch as a music video, video clip, movie trailer, or individual stillphotographs. It is believed that there is no mechanism in the prior artto extract individual frames and concurrent with the extract, provideproof of authenticity or provenance of the extracted frames and metadataabout the extracted frames.

Within this description, the terms engine or machine means a collectionof hardware, which may be augmented by firmware and/or software, thatperforms the described functions. An engine may typically be designedusing a hardware description language (HDL) that defines the engineprimarily in functional terms. The HDL design may be verified using anHDL simulation tool. The verified HDL design may then be converted intoa gate netlist or other physical description of the engine in a processcommonly termed synthesis. The synthesis may be performed automaticallyusing a synthesis tool. The gate netlist or other physical descriptionmay be further converted into programming code for implementing theengine in a programmable device such as PLD, FPGA or PLA. The gatenetlist or other physical description may be converted into processinstructions and masks for fabricating the engine within an ASIC.

Some technologies described for the frame slicing and packaging machines120 and/or the computing device 200 include units. Within thisdescription, the term unit means a collection of hardware, firmware,and/or software, which may be on a larger scale than an engine. Forexample, a unit may contain multiple engines, some of which may performsimilar functions in parallel. The terms engine and unit do not implyany physical separation or demarcation. All or portions of one or moreunits and/or engines may be collocated on a common card, such as anetwork card, or within a common FPGA, ASIC, or other circuit device.

Some technologies described for the physical print production andpackaging machine 520 and/or the computing device 600 include units.Within this description, the term unit means a collection of hardware,firmware, and/or software, which may be on a larger scale than anengine. For example, a unit may contain multiple engines, some of whichmay perform similar functions in parallel. The terms engine and unit donot imply any physical separation or demarcation. All or portions of oneor more units and/or engines may be collocated on a common card, such asa network card, or within a common FPGA, ASIC, or other circuit device.

Some technologies described for the frame slicing and packaging machine920 and/or the computing device 1000 include units. Within thisdescription, the term unit means a collection of hardware, firmware,and/or software, which may be on a larger scale than an engine. Forexample, a unit may contain multiple engines, some of which may performsimilar functions in parallel. The terms engine and unit do not implyany physical separation or demarcation. All or portions of one or moreunits and/or engines may be collocated on a common card, such as anetwork card, or within a common FPGA, ASIC, or other circuit device.

Some technologies described for the frame slicing and packaging machine2320, enhanced image production and packaging machine 2720 and/or thecomputing device 2400 include units. Within this description, the termunit means a collection of hardware, firmware, and/or software, whichmay be on a larger scale than an engine. For example, a unit may containmultiple engines, some of which may perform similar functions inparallel. The terms engine and unit do not imply any physical separationor demarcation. All or portions of one or more units and/or engines maybe collocated on a common card, such as a network card, or within acommon FPGA, ASIC, or other circuit device.

Closing Comments

Throughout this description, the embodiments and examples shown shouldbe considered as exemplars, rather than limitations on the apparatus andprocedures disclosed or claimed. Although many of the examples presentedherein involve specific combinations of method acts or system elements,those acts and those elements may be combined in other ways toaccomplish the same objectives. With regard to flowcharts, additionaland fewer steps may be taken, and the steps as shown may be combined orfurther refined to achieve the methods described herein. Acts, elementsand features discussed only in connection with one embodiment are notintended to be excluded from a similar role in other embodiments.

As used herein, “plurality” means two or more. As used herein, a “set”of items may include one or more of such items. As used herein, whetherin the written description or the claims, the terms “comprising”,“including”, “carrying”, “having”, “containing”, “involving”, and thelike are to be understood to be open-ended, i.e., to mean including butnot limited to. Only the transitional phrases “consisting of” and“consisting essentially of”, respectively, are closed or semi-closedtransitional phrases with respect to claims. Use of ordinal terms suchas “first”, “second”, “third”, etc., in the claims to modify a claimelement does not by itself connote any priority, precedence, or order ofone claim element over another or the temporal order in which acts of amethod are performed, but are used merely as labels to distinguish oneclaim element having a certain name from another element having a samename (but for use of the ordinal term) to distinguish the claimelements. As used herein, “and/or” means that the listed items arealternatives, but the alternatives also include any combination of thelisted items.

It is claimed:
 1. An apparatus comprising a non-volatilemachine-readable medium storing a program having instructions which whenexecuted by a processor will cause the processor to manufacture aninventory of image products from frames of a digital video work, eachframe comprising a still image in sequence in the work, the digitalvideo work including audio which corresponds to objects in the stillimages in the digital video work from which the audio is produced, theinstructions of the program for: detecting objects in each frame'simage; recognizing the objects; assigning metadata to the objects, theobject metadata linking audio from the digital video work to thecorresponding object in the frame's image which produces the audio; foreach frame, generating at least one cryptographic hash of the objectmetadata; writing the hash to a node of a transaction processingnetwork.
 2. The apparatus of claim 1 further comprising assigningmetadata to each frame of the digital video work, the frame metadataincluding an identification of the work, provenance of the work, and anidentification of the frame within the work; and for each frame,generating at least one other cryptographic hash of the frame'smetadata.
 3. The apparatus of claim 1 wherein the digital video work isa whole movie.
 4. The apparatus of claim 1 wherein the digital videowork is a portion of a movie.
 5. The apparatus of claim 1 wherein atleast one recognized object is at least one of a person, an animal or agood.
 6. The apparatus of claim 5 wherein metadata of a person comprisesan actor's name.
 7. The apparatus of claim 1, wherein the objectmetadata includes spoken lines and sounds.
 8. The apparatus of claim 1wherein the transaction processing network is a blockchain ledger. 9.The apparatus of claim 1 further comprising adding a watermark to thehash value before it is written to the node.
 10. The apparatus of claim1 wherein at least some of the metadata links objects across frames. 11.A process for manufacturing an inventory of image products from framesof a digital video work, each frame comprising a still image in sequencein the work, the digital video work including audio which corresponds toobjects in the still images in the digital video work from which theaudio is produced, the process comprising: detecting objects in eachframe's image; recognizing the objects; assigning metadata to theobjects, the object metadata linking audio from the digital video workto the corresponding object in the frame's image; for each frame,generating at least one cryptographic hash of the object metadata;writing the hash to a node of a transaction processing network.
 12. Theprocess of claim 11 further comprising assigning metadata to each frameof the digital video work, the frame metadata including anidentification of the work, provenance of the work, and anidentification of the frame within the work; and for each frame,generating at least one other cryptographic hash of the frame'smetadata.
 13. The process of claim 11 wherein the digital video work isa whole movie.
 14. The process of claim 11 wherein the digital videowork is a portion of a movie.
 15. The process of claim 11 wherein atleast one recognized object is at least one of a person, an animal or agood.
 16. The process of claim 15 wherein metadata of a person comprisesan actor's name.
 17. The process of claim 11, wherein the objectmetadata includes spoken lines and sounds.
 18. The process of claim 11wherein the transaction processing network is a blockchain ledger. 19.The process of claim 11 further comprising adding a watermark to thehash value before it is written to the node.
 20. The process of claim11, wherein at least some of the metadata links objects across frames.